WWC Intervention Report 


U.S. DEPARTMENT OF EDUCATION 

What Works Clearinghouse :*ies«aasar«_ 

Elementary School Math 


September 2010 


Accelerated Math™ 


Program Description 1 2 

Accelerated Math™, published by Renaissance Learning, is 
a software tool used to customize assignments and monitor 
progress in mathematics for students in grades 1-12. 2 Acceler- 
ated Math ™ creates individualized assignments that align with 
state standards and national guidelines, scores student work, 

and generates formative feedback through reports for teachers 
and students. The software can be used in conjunction with the 
existing math curriculum to add practice components and aid 
teachers in differentiating instruction via the program’s progress- 
monitoring data. 


Research 3 One study of Accelerated Math™ that falls within the scope of 
the Elementary School Math review protocol meets What Works 
Clearinghouse (WWC) evidence standards, and two studies meet 
WWC evidence standards with reservations. The three studies 
included 2,179 students from grades 2-5 in over 60 schools 


across multiple states. 4 Studies included in this review assess the 
effectiveness of the first edition of Accelerated Math™. 

Based on these three studies, the WWC considers the extent 
of evidence tor Accelerated Math™ on elementary school stu- 
dents to be medium to large for math achievement. 


Effectiveness Accelerated Math™ was found to have mixed effects on math achievement for elementary school students. 

Math achievement 

Rating of effectiveness Mixed effects 

Improvement index 5 Average: +7 percentile points 

Range: +2 to +10 percentile points 


1. The descriptive information for this program was obtained from publicly available sources: the program’s website (http://www.renlearn.com/am/, down- 
loaded June 2010) and the three studies that met WWC standards either with or without reservations: Nunnery and Ross (2007), Ysseldyke and Bolt 
(2007), and Ysseldyke and Tardrew (2007). The WWC requests developers to review the program description sections for accuracy from their perspec- 
tive. Further verification of the accuracy of the descriptive information for this program is beyond the scope of this review. The literature search reflects 
documents publicly available by August 2008. 

2. This review refers to studies of Accelerated Math™ in kindergarten through fifth grade. Studies of Accelerated Math™ conducted in sixth through 
twelfth grades were out of the scope of the Elementary School Math protocol. 

3. The studies in this report were reviewed using WWC Evidence Standards, Version 1.0 (see the WWC Standards), as described in protocol Version 1.1. 

4. The evidence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 

5. These numbers show the average and range of student-level improvement indices for all findings across the studies. 
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Additional program Developer and contact 

information Renaissance Learning developed and distributes Accelerated 
Math™. Address: PO Box 8036, Wisconsin Rapids, Wl 54495- 
8036. Email: answers@renlearn.com. Web: http://www.renlearn. 
com/am/. Telephone: (800) 338-4204. 

Scope of use 

The first edition of Accelerated Math™ was released in 1998. In 
2008, Renaissance Learning released the Accelerated Math™ 
Second-Edition Libraries, which included a revised scope and 
sequence for grades 1-8, algebra 1, and geometry. According to 
the developers, more than 30,000 schools nationwide use Accel- 
erated Math™ and other Renaissance Learning math programs. 

Teaching 

The Accelerated Math™ software can be used with existing 
textbooks and instructional methods for students in grades 1-12 
to add practice assignments and progress monitoring to the 
existing curriculum. Students are placed into grade-level libraries 
in Accelerated Math™ based on either teacher discretion or their 
performance on a norm-referenced, standardized measure of 
general math achievement. After instruction on a math objective, 
teachers can use the software to create individualized practice 
assignments for students. Students then record their answers 


through handheld responders or on forms that are scanned into 
the computer. After scoring the assignment, the software gener- 
ates a report showing student progress in mastering the objec- 
tive, as well as information about items answered correctly and 
incorrectly. Teachers also receive student- and classroom-level 
reports. After reviewing students’ progress, teachers can adjust 
instruction for the entire class, for small groups of students 
struggling with similar objectives, or for individual students as 
needed. Accelerated Math™ generates future assignments 
based on a student’s performance on previous assignments. 

Cost 6 

Accelerated Math™ can be purchased as a school package or 
a single classroom package. The Accelerated Math™ Enterprise 
Edition, a school package, is available for a one-time school fee, 
plus an annual fee per student. The Enterprise license includes 
nine hours of web-based professional development for teachers, 
content libraries for grade 1 math through calculus, unlimited 
technical support, software updates, and hosting of the software 
for the first year. After the first year, additional web hosting costs 
are incurred. The cost of an optical scanner (needed to grade 
student assignments) is not included. 


Research Thirty-two studies reviewed by the WWC investigated the effects 
of Accelerated Math™. One study (Ysseldyke & Bolt, 2007) is 
a randomized controlled trial that meets WWC evidence stan- 
dards. Two studies (Nunnery & Ross, 2007; Ysseldyke & Tardrew, 
2007) are quasi-experimental designs that meet WWC evidence 
standards with reservations. The remaining 29 studies do not 
meet either WWC evidence standards or eligibility screens 

Meets evidence standards 

Ysseldyke and Bolt (2007) conducted a randomized controlled 
trial of Accelerated Math™ involving schools that had requested 


a price quote for Accelerated Math™ but had not purchased 
the software. In exchange for participation in the study, schools 
were offered Accelerated Math™ software, support materials, 
technical support, and professional development by an Acceler- 
ated Math™ consultant. Each elementary school had to have at 
least three teachers in a given grade to be eligible. Within each 
participating elementary school, teachers in each grade were 
randomly assigned to either the treatment group, the control 
group, or a third group that began implementing Accelerated 
Math™ halfway through the school year. The latter group is 
not utilized in this report. The authors assessed the impact of 


6. Specific pricing is unavailable on the developer’s website. Instead, contact information is listed to be able to obtain a direct quote. 
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Research (continued) Accelerated Math™ using the STAR Math and Terra Nova exams. 

The study sample included students in grades 2-8, but only 
those in grades 2-5 are relevant to this Elementary Math review. 
The elementary school analysis sample included approximately 
700 students (analysis samples varied across outcomes) in 
40 classrooms in five schools across four states. Treatment 
classrooms were assigned to be taught using Accelerated 
Math™ as an integrated addition to the existing math curriculum. 
The existing curricula included: Harcourt Math or Silver Burdett 
Math in Alabama, Houghton Mifflin Math Central in Florida and 
South Carolina, and Sharon Wells Math or Harcourt Math in 
Texas. Control classrooms, which were in the same schools as 
the treatment classrooms, were assigned to be taught using the 
existing curriculum without Accelerated Math™. In practice, the 
Accelerated Math™ program was not implemented for approxi- 
mately 40% of students in grades 2-8 in the initial treatment 
sample; the authors did not report the implementation percent- 
age for the elementary school analysis sample. 

Meets evidence standards with reservations 

Nunnery and Ross (2007) conducted a quasi-experiment to 
assess the impact of the School Renaissance program— a 
comprehensive school reform model, which includes the 
Accelerated Math™ and Accelerated Reader programs— on 
math achievement in a suburban Texas school district. Although 
supplemented by a professional development component known 
as Math Renaissance, the program’s key math component was 
Accelerated Math™. Nine treatment elementary schools from a 
single school district were matched to nine comparison elemen- 
tary schools across multiple districts based on the Texas Educa- 
tion Association’s Academic Excellence Indicator System (AEIS). 
The AEIS groups each school with 40 similar schools based on 
their percentage of African-American, Hispanic, Caucasian, eco- 
nomically disadvantaged, and limited English proficient students, 
as well as student mobility rates as determined by cumulative 
attendance. From the list of 40 similar schools, the most similar 


school on percent of economically disadvantaged students and 
base year accountability rating (low performing, acceptable, 
recognized, or exemplary) was matched to the treatment school, 
with preference given to those schools that did not implement 
Accelerated Math™ or other components of School Renais- 
sance. The treatment schools used Accelerated Math™ as their 
primary math curriculum, but may have supplemented with 
other materials. The comparison school curriculum is unknown. 7 
Although the study sample included students in grades 3-8, only 
students in grades 3-5 are relevant to this review. The analysis 
sample included 865 students in 18 elementary schools (416 
students in nine treatment schools and 449 in nine comparison 
schools) who were pretested in the 3rd grade, received Acceler- 
ated Math™ in the 4th and 5th grades, and have outcomes 
averaged across the 4th and 5th grades. Math achievement was 
measured by the Texas Learning Index math scores obtained 
from the Texas Assessment of Academic Skills. 

Ysseldyke and Tardrew (2007) implemented a classroom 
matched-pairs quasi-experimental design to assess the impact 
of Accelerated Math ™ on the STAR Math test. While the study 
was intended to be a random assignment study, with school 
principals randomly assigning classrooms to treatment or 
comparison conditions, the authors reported that in some cases 
the assignment process had not been random. Thus, the WWC 
reviewed the study as a quasi-experimental design. The total 
study sample (grades 3-10) included 2,397 students in 125 
classrooms in 27 schools in 24 states. Results are reported for 
grade 4, which included 614 students (303 treatment and 311 
comparison). The WWC did not report results for grades 3 and 5 
because the treatment and comparison groups were sufficiently 
different at baseline to require a statistical control for pretest 
differences (see the Elementary School Math protocol for more 
detail). While the authors use a gain score to account for differ- 
ences in baseline test scores, the WWC requires a covariate- 
adjusted estimate for statistical control. Teachers assigned to 
the Accelerated Math™ treatment group were asked to use 


7. Curriculum information for the treatment and comparison groups was clarified by the study authors. 
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Research (continued) the program with their existing math curriculum for the spring 
semester. Comparison group teachers, drawn from the same 
schools, only used the existing math curriculum. 

Extent of evidence 

The WWC categorizes the extent of evidence in each domain 
as small or medium to large (see the WWC Procedures and 


Standards Handbook, Appendix G). The extent of evidence takes 
into account the number of studies and the total sample size 
across the studies that meet WWC evidence standards with or 
without reservations. 8 

The WWC considers the extent of evidence for Accelerated 
Math™ on elementary school students to be medium to large for 
math achievement. 


Effectiveness Findings 

The WWC review of interventions for Elementary School Math 
addresses student outcomes in math achievement. The findings 
below present the authors’ estimates and WWC-calculated 
estimates of the size and statistical significance of the effects of 
Accelerated Math™ on elementary school students. 8 9 

Ysseldyke and Bolt (2007) examined two math achievement 
outcomes: the STAR Math and Terra Nova math subtests. The 
authors reported a statistically significant positive effect for 
one outcome (STAR Math) and no statistically significant effect 
for the other (Terra Nova). 10 In WWC calculations, for both out- 
comes, the effects were neither statistically significant nor large 
enough to be considered substantively important according to 
WWC criteria (an effect size greater than 0.25). 

Nunnery and Ross (2007) reported a positive and statistically 
significant effect of Accelerated Math™ on overall math achieve- 
ment based on the Texas Learning Index math scores. In WWC 
calculations, this finding was neither statistically significant nor 
substantively important according to WWC criteria. 

Ysseldyke and Tardrew (2007) reported a positive and statisti- 
cally significant effect of Accelerated Math™ for grades 3, 4, and 


5 on overall math achievement based on STAR Math scale scores. 
In WWC calculations for grade 4, the findings were not statistically 
significant, but were large enough to be considered substantively 
important according to WWC criteria. For grades 3 and 5, the 
WWC could not confirm the findings because the treatment and 
comparison groups were not equivalent at baseline, and the 
results did not employ a statistical control for pretest differences. 

In summary, the WWC reviewed findings from three studies 
that contained math achievement outcomes. One of the studies 
found substantively important positive effects, while the other 
two studies showed indeterminate effects. 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effectiveness 
takes into account four factors: the quality of the research design, 
the statistical significance of the findings, the size of the difference 
between participants in the intervention and the comparison condi- 
tions, and the consistency in findings across studies (see the WWC 
Procedures and Standards Handbook, Appendix E). 


8. The extent of evidence categorization was developed to tell readers how much evidence was used to determine the intervention rating, focusing on the 
number and size of studies. Additional factors associated with a related concept— external validity, such as the students’ demographics and the types 
of settings in which studies took place— are not taken into account for the categorization. Information about how the extent of evidence rating was 
determined tor Accelerated Math™ is in Appendix A5. 

9. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within class- 
rooms or schools and for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate 
the statistical significance, see WWC Procedures and Standards Handbook, Appendix C for clustering and WWC Procedures and Standards Handbook, 
Appendix D for multiple comparisons. In the case of Ysseldyke and Bolt (2007), corrections for clustering and multiple comparisons were needed, so 
the significance levels may differ from those reported in the original study. In the cases of Nunnery and Ross (2007) and Ysseldyke and Tardrew (2007), 
corrections for clustering were needed, so the significance levels may differ from those reported in the original studies. 

10. The study authors provided the WWC with findings relevant for the Elementary School Math review. 
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The average improvement index for math achievement is +7 
percentile points across the three studies, with a range of +2 
to +10 percentile points across findings. 

Summary 

The WWC reviewed 32 studies on Accelerated Math™. One of 
these studies meets WWC evidence standards; two studies meet 
WWC evidence standards with reservations; the remaining 29 
studies do not meet either WWC evidence standards or eligibility 
screens. Based on the three studies, the WWC found mixed 
effects in math achievement for elementary school students. 

The conclusions presented in this report may change as new 
research emerges. 
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Appendix 


Appendix A1.1 

Study characteristics: Ysseldyke & Bolt, 2007 

Characteristic 

Description 

Study citation 

Ysseldyke, J., & Bolt, D. M. (2007). Effect of technology-enhanced continuous progress monitoring on math achievement. School Psychology Review, 36(3), 453-467. 

Participants 

The initial study sample included 3,309 students from 133 classrooms in nine schools, representing eight school districts in eight states. Sample students were in grades 2-8 
during the 2003-04 school year. In the initial study sample, 38% of the students were Hispanic, 28% were African-American, 24% were Caucasian, 1% were Asian, 0% were 
Native American, and 8% were not specified. The baseline sample for grades 2-5 was 896 students (441 treatment and 455 control) in 40 classrooms (20 treatment and 
20 control). The results here are drawn from the test-takers in the 40 elementary school classrooms included in the analysis — 587 students took the STAR Math test (315 
treatment, 272 control) and 700 took the Terra Nova test (355 treatment, 345 control). Postattrition treatment and control groups were equivalent on pretests at baseline. 

Setting 

The elementary school sample analyzed here comprises five schools in four states: Alabama, Florida, South Carolina, and Texas. 

Intervention 

Students were taught by teachers using the Accelerated Math ™ program during the 2003-04 school year. Teachers assigned to the treatment group were asked to use 
Accelerated Math™ with their regular math curriculum. The existing curricula included: Harcourt Math or Silver Burdett Math in Alabama, Houghton Mifflin Math Central in Florida 
and South Carolina, and Sharon Wells Math or Harcourt Math in Texas. In practice, the program was not implemented for approximately 40% of grade 2-8 students in the initial 
treatment group; the authors did not report the percentage of grade 2-5 students in the treatment group of the analysis sample that did not participate in Accelerated Math™. 

Comparison 

Students in the control group were taught using the same set of math curricula as the treatment group, but without the addition of Accelerated Math™. 

Primary outcomes 
and measurement 

Participating students were pretested in October 2003 and posttested in May 2004 using two nationally normed, standardized tests (STAR Math and Terra Nova) for math 
achievement. Students in the treatment and control groups were compared using a linear regression analysis in which posttest scores were regressed on pretest scores. In 
the paper, the authors include school level dummies and interactions between the school and the treatment group in their analysis of outcomes for students in grades 2-8. 
The results presented in this report related to students in grades 2-5 (provided to the WWC by the author) include only controls for pretests. For a more detailed description 
of these outcome measures, see Appendix A2. 

Staff/teacher training 

Teachers in the intervention group were trained to use Accelerated Math™. During the school year, teachers using Accelerated Math™ received three to five visits from 
a Renaissance Learning math consultant, who guided teachers on how to improve their use of the program. Teachers also had unlimited access to technical support. 
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Appendix A1.2 Study characteristics: Nunnery & Ross, 2007 


Characteristic 

Description 

Study citation 

Nunnery, J., & Ross, S. M. (2007). The effects of the School Renaissance program on student achievement in reading and mathematics. Research in the Schools, 74(1), 
40-59. 

Participants 

The analysis sample included 865 students (41 6 treatment, 449 comparison) from 18 elementary schools (nine treatment and nine comparison). These students were in grade 
5 during the 2001-02 school year. Students in the analysis sample remained in the same school and had matched data available for three consecutive years (the 1999-2000 
to 2001-02 school years). Characteristics of the student sample varied across the 18 schools. Between 0% and 59% in each school qualified for free or reduced-price lunch. 
Between 0% and 35% of students in each school were limited English proficient, and the proportion of students who were Caucasian ranged from 25% to 95%. 

Setting 

The treatment group schools came from one suburban school district in Texas. Comparison schools came from other school districts with similar populations of students 
in Texas. 

Intervention 

In the 2000-01 school year, schools in the treatment group began implementing School Renaissance, a comprehensive school reform model that includes Accelerated 
Math™. Students in the treatment group experienced two years of the Accelerated Math™ program as their primary mathematics curriculum. Treatment schools may have 
supplemented with other materials. 

Comparison 

Schools in the comparison condition were from Texas school districts that had not implemented the full School Renaissance package. It is possible that some elements 
of School Renaissance (e.g., Accelerated Math™) were present in the comparison schools; however, the comparison group curriculum is unknown. 

Primary outcomes 
and measurement 

The study used the Texas Learning Index math scores (based on the Texas Assessment of Academic Skills); for the grade 5 cohort, program comparisons were based 
on average transformed scores for grades 4 and 5 from 2001 and 2002. For a more detailed description of this outcome measure, see Appendix A2. 

Staff/teacher training 

A Renaissance coach conducted an initial training seminar and provided ongoing assistance to teachers. 
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Appendix A1.3 Study characteristics: Ysseldyke & Tardrew, 2007 


Characteristic 

Description 

Study citation 

Ysseldyke, J., & Tardrew, S. (2007). Use of a progress monitoring system to enable teachers to differentiate mathematics instruction. Journal of Applied School Psychology, 
24(1), 1-28. 

Participants 

The initial study sample included 2,397 students (1,319 treatment and 1,078 comparison) in grades 3-10 during the 2001-02 school year. Students were drawn from 125 
classrooms (67 treatment and 58 comparison) in 47 schools in 24 states. The elementary school analysis sample in this review included 1,680 students (869 treatment and 
811 comparison) in grades 3-5. The grade 3 and grade 5 samples had large differences in baseline test scores that were not controlled for in the analysis. Therefore, only the 
grade 4 results for 614 students (303 treatment and 311 comparison) are reported. 

Setting 

The study was conducted in 47 schools in 24 states (Alabama, Arkansas, California, Georgia, Idaho, Illinois, Indiana, Iowa, Kansas, Massachusetts, Michigan, Minnesota, 
Missouri, Montana, New Mexico, Ohio, Oklahoma, Oregon, Pennsylvania, Tennessee, Texas, Virginia, Washington, and Wisconsin). The authors did not report whether all 
schools and states were represented in the elementary school sample. 

Intervention 

Students were taught by teachers using Accelerated Math™ during the spring semester of the 2001-02 school year. Teachers assigned to the Accelerated Math™ 
treatment group were asked to use the program with their existing math curriculum. 

Comparison 

Comparison classrooms were drawn from the same schools as treatment classrooms. Teachers assigned to the comparison group used their usual math curriculum and 
practices. 

Primary outcomes 
and measurement 

Students were pretested in January 2002 and posttested in May 2002 using STAR Math, a computer adaptive math achievement test. For a more detailed description of 
this outcome measure, see Appendix A2. 

Staff/teacher training 

Intervention teachers participated in a one-day training session conducted by Renaissance Learning. The training was designed to familiarize teachers with Accelerated 
Math™ and to guide them in integrating it into curriculum and instruction. Of 68 treatment group teachers in the full sample (grades 3-8), 66 attended the training. 


Appendix A2 Outcome measures for the math achievement domain 


Outcome measure 

Description 

Terra Nova 
mathematics subtest 

The Terra Nova subtest is a national norm-referenced test that assesses academic performance in math. 

Texas Learning Index 
math scores (based on 
the Texas Assessment 
of Academic Skills) 

The Texas Assessment of Academic Skills (TAAS) is a criterion-referenced standardized state test that measures problem-solving and critical-thinking skills. The Texas 
Learning Index (TLI) is an outcome metric, based on student performance on the TAAS, which allows for comparisons between administrations and between grades. The TLI 
has a common interpretation across grades: a score of 70 or above indicates the student performed at or above grade-level expectations. A student receiving the same score 
at consecutive grade levels made one year of academic progress. Analyses in the study were based on a transformation of the TLI that was conducted to give the measure a 
desirable statistical property (a normal distribution). 

STAR Math assessment 

STAR Math is a computer-adaptive math test that assesses math skills. It combines computation and numeration items with word problems, estimation, statistics, charts 
and graphs, geometry, measurement, and algebra. STAR scores can appear as scaled scores or normal curve equivalent values. STAR Math was developed by Renaissance 
Learning. 
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Appendix A3 Summary of study findings included in the rating for the math achievement domain 1 





Authors’ findings from the study 







Mean outcome 
(standard deviation) 2 


WWC calculations 


Outcome measure 

Study 

sample 

Sample size 
(students) 

Accelerated 

Math ™ Comparison 

group group 

Mean 

difference 3 
(Accelerated 
Math ™ 

- comparison) 

Statistical 

Effect significance 5 

size 4 (at a = 0.05) 

Improvement 

index 6 


STAR Math normal curve 
equivalent scores 

Grades 2-5 

587 

Ysseldyke & Bolt, 2007 7 

47.1 2 s 44. 72 9 

(21.40) (24.05) 

2.40 

0.11 

ns 

+4 

Terra Nova normal curve 
equivalent scores 

Grades 2-5 

700 

46.59 s 

(18.23) 

45. 43 9 
(19.89) 

1.16 

0.06 

ns 

+2 

Average for math achievement (Ysseldyke & Bolt, 2007) 10 




0.08 

ns 

+3 




Nunnery & Ross, 2007 7 





2001 and 2002 transformed 

Grade 5 cohort 

865 

1.26 11 

1.21 

0.05 

0.17 

ns 

+7 

Texas Learning Index scores 



(0.28) 

(0.30) 





Average for math achievement (Nunnery & Ross, 2007) 10 




0.17 

ns 

+7 




Ysseldyke & Tardrew, 2007 7 





STAR Math scale scores 

Grade 4 

614 

686.50 12 

665.22 13 

21.28 

0.25 

ns 

+10 




(85.74) 

(85.46) 





Average for math achievement (Ysseldyke & Tardrew, 2007) 10 




0.25 

ns 

+10 

Domain average for math achievement across all studies 10 




0.17 

na 

+7 


ns = not statistically significant 
na = not applicable 


1. This appendix reports findings considered for the effectiveness rating and the average improvement indices for the math achievement domain. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

4. For an explanation of the effect size calculation, see WWC Procedures and Standards Handbook, Appendix B. 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results for the intervention group. 

7. The level of statistical significance was reported by the study authors or, when necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple compari- 
sons. For an explanation, see the WWC Tutorial on Mismatch. For the formulas the WWC used to calculate the statistical significance, see WWC Procedures and Standards Handbook, Appendix 
C for clustering and WWC Procedures and Standards Handbook, Appendix D for multiple comparisons. In the case of Ysseldyke and Bolt (2007), corrections for clustering and multiple com- 
parisons were needed, so the significance levels may differ from those reported in the original study. In the cases of Nunnery and Ross (2007) and Ysseldyke and Tardrew (2007), corrections for 
clustering were needed, so the significance levels may differ from those reported in the original studies. 

(continued) 


WWC Intervention Report Accelerated Math™ 


September 2010 


12 




Appendix A3 Summary of study findings included in the rating for the math achievement domain (continued) 

8. The intervention group means from Ysseldyke and Bolt (2007) differ from those presented in the paper. The WWC calculated the intervention group means by adding the impact of the program, 
derived from a regression that include a control for pretest differences, to the unadjusted comparison group means. The study authors provided the WWC with the impact, unadjusted compari- 
son group mean, and standard deviations for both groups only for elementary schools. 

9. The comparison group mean from Ysseldyke and Bolt (2007) is unadjusted. 

10. The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated 
from the average effect sizes. 

11. Nunnery and Ross (2007, pp. 45-46) computed a transformation of the Texas Learning Index score to give the measure a normal distribution and allow for an analysis of covariance. 

12. The intervention group mean from Ysseldyke and Tardrew (2007) equals the unadjusted comparison group mean plus the difference in gains between the treatment and comparison groups from 
pretest to posttest. 

13. The comparison group mean from Ysseldyke and Tardrew (2007) is unadjusted. 
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Appendix A4 Accelerated Math ™ rating for the math achievement domain 

The WWC rates an intervention’s effects for a given outcome domain as positive, potentially positive, mixed, no discernible effects, potentially negative, or negative. 1 

For the outcome domain of math achievement, the WWC rated Accelerated Math ™ as having mixed effects for elementary school students. The remaining ratings 
(no discernible effects, potentially negative effects, negative effects) were not considered, as Accelerated Math™ was assigned the highest applicable rating. 

Rating received 

Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect, and at least one study showing a statistically significant 
or substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important positive effect. 

Not met. One study showed substantively important positive effects. No studies showed statistically significant or substantively important 
negative effects. 

OR 

• Criterion 2: At least one study showing a statistically significant or substantively important effect, and more studies showing an indeterminate effect than showing 
a statistically significant or substantively important effect. 

Met. One study showed substantively important positive effects, and two studies showed indeterminate effects. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. No studies showed statistically significant positive effects. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. No studies showed statistically significant or substantively important negative effects. 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Met. One study showed substantively important positive effects. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect, and fewer or the same number of studies showing indetermi- 
nate effects than showing statistically significant or substantively important positive effects. 

Not met. No studies showed negative effects, but more studies showed indeterminate effects than positive effects. 

1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. For a complete description, see the WWC Procedures and Standards Handbook, Appendix E. 
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Appendix A5 Extent of evidence by domain 





Sample size 


Outcome domain 

Number of studies 

Schools 

Students 

Extent of evidence 1 

Math achievement 

3 

61 2 

2,179 3 

Medium to large 


1. A rating of “medium to large” requires at least two studies and two schools across studies in one domain and a total sample size across studies of at least 350 students or 14 classrooms. 
Otherwise, the rating is “small.” For more details on the extent of evidence categorization, see the WWC Procedures and Standards Handbook, Appendix G. 

2. For Ysseldyke and Tardrew (2007), the WWC was unable to determine the exact number of schools that were included in the analysis sample for the grades presented in this report. 

3. For Ysseldyke and Bolt (2007), this presumes that the students who took the STAR Math test are a subset of those who took the Terra Nova. Otherwise, the number of students could be larger 
than 2,179. 
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